Mobile terminal and control method therefor

ABSTRACT

Disclosed are a mobile terminal and a control method therefor. The mobile terminal, according to one embodiment of the present invention, comprises: a camera provided to a main body; a display unit for displaying a preview image for the photographing of the camera; and a control unit for detecting the entering into of a subject information search mode while the preview image is being displayed, and in the information search mode, detecting the selection of one search option among a plurality of search options for a subject. In addition, on the basis of a touch input applied to the preview image, the control unit photographs the subject, including an object to which the selected search option is to be applied, and displays, on the display unit, an image of the photographed subject along with a screen where the selected search option is applied.

BACKGROUND 1. Technical Field

The present disclosure relates to a mobile terminal having a camera anda control method thereof.

2. Description of the Related Art

Terminals may be divided into mobile/portable terminals and stationaryterminals according to their mobility. Furthermore, mobile terminals maybe divided into handheld terminals and vehicle mounted terminalsaccording to whether or not it can be directly carried by a user.

The functions of mobile terminals have been diversified. For example,the functions may include data and voice communication, photographingand video shooting through a camera, voice recording, playing a musicfile through a speaker system, and displaying an image or video on adisplay module. Some terminals further include an electronic game playfunction or perform a multimedia player function. In particular, inrecent years, mobile terminals may receive multicast signals thatprovide visual content such as broadcast, video or television programs.

As it becomes multifunctional, for example, such a terminal is allowedto capture still images or moving images, play music or video files,play games, receive broadcast and the like, so as to be implemented asan integrated multimedia player.

In recent years, as the use of the camera in the terminal is furtherincreased and the pixels of the applied camera are further improved,various functions or shooting modes as in a professional camera areprovided. For such an example, a service option that analyzes an imageof a subject and provides an appropriate search function, and the likefor user's convenience has appeared.

However, an option different from the user's intention may be providedsince this automatically analyzes an image of the captured subject andprovides an option determined to be appropriate according to thesubject. This may be due to an error in the image analysis of thesubject, a difference in the application range of the option, or due tothe very diverse preferences of users, and there is a limit inreflecting the user's taste through image analysis alone.

In this case, the user still has the inconvenience of having to go backto the previous step and manually set or change the desired serviceoption or application target. In addition, there is a loss of time andresources due to unnecessary image analysis.

SUMMARY

Accordingly, one aspect according to an embodiment of the presentdisclosure is to provide a mobile terminal and a control method thereofcapable of accurately specifying a user's desired option from a subjectto be captured, and then providing various search functions through theimage analysis.

Furthermore, another aspect according to an embodiment of the presentdisclosure is to provide a mobile terminal and a control method thereofcapable of acquiring a desired search result based on the information ofthe captured image by simply capturing an object in reality through acamera without going through a separate search window or a web search.

In addition, another aspect according to an embodiment of the presentdisclosure is to provide a mobile terminal and a control method thereofcapable of accurately specifying a user's desired part of objects when aplurality of objects are included in a subject to be captured, andselectively receiving only search results for the desired part.

In order to achieve the foregoing objectives, a mobile terminalaccording to an embodiment of the present disclosure may include acamera provided in a main body; a display module that displays a previewimage for the shooting of the camera; and a controller that detects theentry of a subject into an information search mode while the previewimage is being displayed, and detects the selection of one of aplurality of search options for the subject in the information searchmode, wherein the controller captures a subject including an object towhich the selected search option is to be applied, and displays an imageof the captured subject along with an application screen of the selectedsearch option on the display module.

Furthermore, in one embodiment, a plurality of items corresponding to anexecution result of the selected search option for the object may bedisplayed on the application screen of the selected search option, and apage screen linked to the selected item may be displayed in response tothe selection of any one of the displayed plurality of items.

Furthermore, in one embodiment, when a touch input is applied to abackground region of the preview image subsequent to the selection of asearch option, the selected search option may be applied to all objectsincluded in the preview image.

Furthermore, in one embodiment, when a back execution command isreceived while the application screen of the selected search option isdisplayed, it may be switched to a screen for changing an object towhich the selected search option is to be applied.

Furthermore, in one embodiment, when a touch input is applied to any oneobject in a preview image including a plurality of objects subsequentthe selection of a search option, the controller may limit theapplication of the selected search option to the remaining objects.

Furthermore, in one embodiment, while the image of the captured subjectis displayed along with the application screen of the selected searchoption, a guide image indicating an object to which the selected searchoption is to be applied may be displayed on the image of the capturedsubject.

In addition, a mobile terminal according to an embodiment of the presentdisclosure may include a camera provided in a main body; a displaymodule that displays a preview image for the shooting of the camera; anda controller that detects the entry of a subject into an informationsearch mode while the preview image is being displayed, and detects theselection of a text search option among a plurality of search optionsfor the subject in the information search mode, wherein the controllerextracts text information to which the text search option is to beapplied from the preview image based on a touch input applied to thepreview image, and switches the preview image to an extraction screendisplayed with the extracted text information, and switches theplurality of search options to a function icon related to the processingof text information displayed on the extraction screen to display thefunction icon.

Furthermore, in one embodiment, the function icon related to theprocessing of the text information may be determined differentlyaccording to the attribute of the text information displayed on theextraction screen.

Furthermore, in one embodiment, an execution screen of a functioncorresponding to the selected function icon for the text information maybe displayed on the display module in response to the selection of anyone of functions icons related to the processing of the textinformation.

Furthermore, in one embodiment, when a touch input is applied to theexecution screen, information displayed on the execution screen may besaved along with the extraction screen.

Furthermore, in one embodiment, in response to the selection of sometexts in the extraction screen while a first group of function icons arebeing displayed, the first group of function icons may be changed to asecond group of function icons related to the attribute of the selectedsome texts.

Furthermore, in one embodiment, a capture image of the preview image maybe stored in a memory while text information is extracted from thepreview image according to a touch input applied to the preview image.

As described above, according to a mobile terminal and a control methodthereof according to an embodiment of the present disclosure, a user mayspecify a search object by simply pointing a camera without having todirectly perform information search related to an object in reality, andquickly and easily acquire his or her desired information search result.Furthermore, a desired option may be accurately specified among aplurality of search options prior to image analysis on a subject,thereby eliminating errors due to automatic selection of a search objectand a search option.

In addition, a desired search result may be acquired by selecting adesired search option from a preview image including a plurality ofobjects, and then selectively specifying any one object from thebeginning or including all of the objects. Moreover, it may be possibleto quickly change a target object or a search option itself even afterchecking the search result.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram for explaining the configuration of a mobileterminal associated with the present disclosure.

FIGS. 1B and 10 are conceptual views in which a mobile terminalassociated with the present disclosure is seen from differentdirections.

FIGS. 2A, 2B, 2C and 2D are conceptual views for explaining a specificoperation according to the execution of a subject information searchmode in a mobile terminal associated with the present disclosure.

FIG. 3 is a representative flowchart for explaining a specific operationaccording to the execution of a subject information search mode in amobile terminal associated with the present disclosure.

FIGS. 4A, 4B, 4C, 4D, 4E and 4F are exemplary views for explaining anoperation of quickly specifying a desired object to execute aninformation search function when a plurality of objects are included inan object to be captured in a mobile terminal associated with thepresent disclosure.

FIGS. 5A, 5B, 5C and 5D are exemplary views for explaining an executionoperation of an image search function related to an image of a subjectin a mobile terminal associated with the present disclosure.

FIGS. 6, 7A, 7B, 7C, 7D, 8A, 8B, 8C, 8D, 9A, 9B, 9C, 9D, 10A and 10B areexemplary views for explaining an operation of extracting text includedin an image of a subject to provide an additional function thereto in amobile terminal associated with the present disclosure.

FIGS. 11A, 11B and 11C are exemplary views for explaining an operationof executing a QR scan function on an image of a subject in a mobileterminal associated with the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, the embodiments disclosed herein will be described indetail with reference to the accompanying drawings, and the same orsimilar elements are designated with the same numeral referencesregardless of the numerals in the drawings and their redundantdescription will be omitted. A suffix “module” and “unit” used forconstituent elements disclosed in the following description is merelyintended for easy description of the specification, and the suffixitself does not give any special meaning or function. In describing thepresent disclosure, moreover, the detailed description will be omittedwhen a specific description for publicly known technologies to which theinvention pertains is judged to obscure the gist of the presentdisclosure. The accompanying drawings are used to help easily understandvarious technical features and it should be understood that theembodiments presented herein are not limited by the accompanyingdrawings. As such, the present disclosure should be construed to extendto any alterations, equivalents and substitutes in addition to thosewhich are particularly set out in the accompanying drawings.

It will be understood that although the terms first, second, etc. may beused herein to describe various elements, these elements should not belimited by these terms. These terms are generally only used todistinguish one element from another.

It will be understood that when an element is referred to as being“connected with” another element, the element can be connected with theother element or intervening elements may also be present. In contrast,when an element is referred to as being “directly connected with”another element, there are no intervening elements present.

A singular representation may include a plural representation unless itrepresents a definitely different meaning from the context.

Terms “include” or “has” used herein should be understood that they areintended to indicate the existence of a feature, a number, a step, aconstituent element, a component or a combination thereof disclosed inthe specification, and it may also be understood that the existence oradditional possibility of one or more other features, numbers, steps,constituent elements, components or combinations thereof are notexcluded in advance.

A mobile terminal described herein may include a mobile phone, a smartphone, a laptop computer, a digital broadcasting terminal, a personaldigital assistant (PDA), a portable multimedia player (PMP), navigation,a slate PC, a tablet PC, an ultrabook, a wearable device (i.e., awatch-type terminal (smartwatch), glasses-type terminal (smart glasses),a head mounted display (HMD), and the like.

A mobile terminal described herein may include a mobile phone, a smartphone, a laptop computer, a digital broadcasting terminal, a personaldigital assistant (PDA), a portable multimedia player (PMP), navigation,a slate PC, a tablet PC, an ultrabook, a wearable device (i.e., awatch-type terminal (smartwatch), glasses-type terminal (smart glasses),a head mounted display (HMD), and the like.

However, it may be easily understood by those skilled in the art thatthe configuration according to the exemplary embodiments of thisspecification can also be applied to stationary terminals such asdigital TV, desktop computers, digital signages, and the like, excludinga case of being applicable only to the mobile terminals.

Referring to FIGS. 1A through 10 , FIG. 1A is a block diagram forexplaining a mobile terminal associated with the present disclosure, andFIGS. 1B and 10 are conceptual views illustrating an example in whichthe mobile terminal associated with the present disclosure is seen fromdifferent directions.

The mobile terminal 100 may include components, such as a wirelesscommunication unit 110, an input unit 120, a sensing unit 140, an outputunit 150, an interface unit 160, a memory 170, a controller 180, a powersupply unit 190 and the like. The components shown in FIG. 1A are notessential for implementing a mobile terminal, and thus the mobileterminal described herein may have more or fewer components than thoselisted above.

In more detail, the wireless communication unit 110 of those componentsmay typically include one or more modules which permit wirelesscommunications between the mobile terminal 100 and a wirelesscommunication system, between the mobile terminal 100 and another mobileterminal 100, or between the mobile terminal 100 and an external server.In addition, the wireless communication unit 110 may include one or moremodules for connecting the mobile terminal 100 to one or more networks.

The wireless communication unit 110 may include at least one of abroadcast receiving module 111, a mobile communication module 112, awireless Internet module 113, a short-range communication module 114, alocation information module 115 and the like.

The input unit 120 may include a camera 121 for inputting an imagesignal, a microphone 122 or an audio input module for inputting an audiosignal, or a user input unit 123 (for example, a touch key, a push key(or a mechanical key), etc.) for allowing a user to input information.Audio data or image data collected by the input unit 120 may be analyzedand processed by a user's control command.

The sensing unit 140 may include at least one sensor which senses atleast one of internal information of the mobile terminal, a surroundingenvironment of the mobile terminal and user information. For example,the sensing unit 140 may include a proximity sensor 141, an illuminationsensor 142, a touch sensor, an acceleration sensor, a magnetic sensor, aG-sensor, a gyroscope sensor, a motion sensor, an RGB sensor, aninfrared (IR) sensor, a finger scan sensor, a ultrasonic sensor, anoptical sensor (for example, refer to the camera 121), a microphone 122,a battery gage, an environment sensor (for example, a barometer, ahygrometer, a thermometer, a radiation detection sensor, a thermalsensor, a gas sensor, etc.), and a chemical sensor (for example, anelectronic nose, a health care sensor, a biometric sensor, etc.). On theother hand, the mobile terminal disclosed herein may utilize informationin such a manner of combining information sensed by at least two sensorsof those sensors.

The output unit 150 may be configured to output an audio signal, a videosignal or a tactile signal. The output unit 150 may include a displaymodule 151, an audio output module 152, a haptic module 153, an opticaloutput unit 154 and the like. The display module 151 may beinter-layered or integrated with a touch sensor to implement a touchscreen. The touch screen may provide an output interface between themobile terminal 100 and a user, as well as functioning as the user inputunit 123 which provides an input interface between the mobile terminal100 and the user.

The interface unit 160 may serve as an interface with various types ofexternal devices connected with the mobile terminal 100. The interfaceunit 160, for example, may include wired or wireless headset ports,external power supply ports, wired or wireless data ports, memory cardports, ports for connecting a device having an identification module,audio input/output (I/O) ports, video I/O ports, earphone ports, or thelike. The mobile terminal 100 may execute an appropriate controlassociated with a connected external device, in response to the externaldevice being connected to the interface unit 160.

In addition, the memory 170 stores data that support various functionsof the mobile terminal 100. The memory 170 is typically implemented tostore data to support various functions or features of the mobileterminal 100. For instance, the memory 170 may be configured to storeapplication programs executed in the mobile terminal 100, data orinstructions for operations of the mobile terminal 100, and the like. Atleast some of those application programs may be downloaded from anexternal server via wireless communication. Some others of thoseapplication programs may be installed within the mobile terminal 100 atthe time of being shipped for basic functions of the mobile terminal 100(for example, receiving a call, placing a call, receiving a message,sending a message, etc.). On the other hand, the application programsmay be stored in the memory 170, installed in the mobile terminal 100,and executed by the controller 180 to perform an operation (or afunction) of the mobile terminal 100.

The controller 180 may typically control an overall operation of themobile terminal 100 in addition to the operations associated with theapplication programs. The controller 180 may provide or processinformation or functions appropriate for a user in a manner ofprocessing signals, data, information and the like, which are input oroutput by the aforementioned components, or activating the applicationprograms stored in the memory 170.

Furthermore, the controller 180 may control at least part of thecomponents illustrated in FIG. 1A, in order to drive the applicationprograms stored in the memory 170. In addition, the controller 180 maydrive the application programs by combining at least two of thecomponents included in the mobile terminal 100 for operation.

The power supply unit 190 may receive external power or internal powerand supply appropriate power required for operating respective elementsand components included in the mobile terminal 100 under the control ofthe controller 180. The power supply unit 190 may include a battery, andthe battery may be an embedded battery or a replaceable battery.

At least part of those elements and components may be combined toimplement operation and control of the mobile terminal or a controlmethod of the mobile terminal according to various exemplary embodimentsdescribed herein. Furthermore, the operation and control or the controlmethod of the mobile terminal may be implemented in the mobile terminalin such a manner of activating at least one application program storedin the memory 170.

Hereinafter, each aforementioned component will be described in moredetail with reference to FIG. 1A, prior to explaining various exemplaryembodiments implemented by the mobile terminal 100 having theconfiguration.

First, the wireless communication unit 110 will be described. Thebroadcast receiving module 111 of the wireless communication unit 110may receive a broadcast signal and/or broadcast associated informationfrom an external broadcast managing entity via a broadcast channel. Thebroadcast channel may include a satellite channel and/or a terrestrialchannel. At least two broadcast receiving modules 111 may be provided inthe mobile terminal 100 to simultaneously receive at least two broadcastchannels or switch the broadcast channels.

The mobile communication module 112 may transmit/receive wirelesssignals to/from at least one of network entities, for example, a basestation, an external terminal, a server, and the like, on a mobilecommunication network, which is constructed according to technicalstandards or transmission methods for mobile communications (forexample, Global System for Mobile communication (GSM), Code DivisionMulti Access (CDMA), Code Division Multi Access 2000 (CDMA2000),Enhanced Voice-Data Optimized or Enhanced Voice-Data Only (EV-DO),Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), HighSpeed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long TermEvolution-Advanced (LTE-A), etc.)

Here, the wireless signals may include audio call signal, video(telephony) call signal, or various formats of data according totransmission/reception of text/multimedia messages.

The wireless Internet module 113 refers to a module for supportingwireless Internet access, and may be built-in or externally installed onthe mobile terminal 100. The wireless Internet module 113 may transmitand/or receive wireless signals via communication networks according towireless Internet technologies.

Examples of such wireless Internet access may include Wireless LAN(WLAN), Wireless-Fidelity (Wi-Fi), Wireless Fidelity Direct (Wi-FiDirect), Digital Living Network Alliance (DLNA), Wireless Broadband(WiBro), World Interoperability for Microwave Access (WiMAX), High SpeedDownlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA),LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), and thelike. The wireless Internet module 113 may transmit/receive dataaccording to at least one wireless Internet technology within a rangeincluding even Internet technologies which are not aforementioned.

From the perspective that the wireless Internet accesses according toWibro, HSDPA, GSM, CDMA, WCDMA, LTE, LTE-A and the like are executed viaa mobile communication network, the wireless Internet module 113 whichperforms the wireless Internet access via the mobile communicationnetwork may be understood as a type of the mobile communication module112.

The short-range communication module 114 denotes a module forshort-range communications. Suitable technologies for implementing theshort-range communications may include BLUETOOTH™, Radio FrequencyIDentification (RFID), Infrared Data Association (IrDA), Ultra-WideBand(UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity(Wi-Fi), Wi-Fi Direct, and the like. The short-range communicationmodule 114 may support wireless communications between the mobileterminal 100 and a wireless communication system, between the mobileterminal 100 and another mobile terminal 100, or between the mobileterminal and a network where another mobile terminal 100 (or an externalserver) is located, via wireless personal area networks. The short-rangecommunication module 114 denotes a module for short-rangecommunications.

Here, the another mobile terminal 100 may be a wearable device, forexample, a smart watch, smart glasses or a head mounted display (HMD),which is able to exchange data with the mobile terminal 100 (or to linkdata with the mobile terminal 100). The short-range communication module114 may sense (recognize) a wearable device, which is able tocommunicate with the mobile terminal), near the mobile terminal 100. Inaddition, when the sensed wearable device is a device which isauthenticated to communicate with the mobile terminal 100 according tothe present disclosure, the controller 180 may transmit at least part ofdata processed in the mobile terminal 100 to the wearable device via theshort-range communication module 114. Hence, a user of the wearabledevice may use the data processed in the mobile terminal 100 on thewearable device. For example, when a call is received in the mobileterminal 100, the user may answer the call using the wearable device.Also, when a message is received in the mobile terminal 100, the usercan check the received message using the wearable device.

The location information module 115 is generally configured to detect,calculate, derive or otherwise identify a position of the mobileterminal. As an example, the location information module 115 includes aGlobal Position System (GPS) module, a WiFi module, or both. Forexample, when the mobile terminal uses the GPS module, a position of themobile terminal may be acquired using a signal sent from a GPSsatellite. As another example, when the mobile terminal uses the Wi-Fimodule, a position of the mobile terminal may be acquired based oninformation associated with a wireless access point (AP) which transmitsor receives a wireless signal to or from the Wi-Fi module. According tothe need, the location information module 115 may perform any functionof the other modules of the wireless communication unit 110 to obtaindata on the location of the mobile terminal. As a module used to acquirethe location (or current location) of the mobile terminal, the locationinformation module 115 may not be necessarily limited to a module fordirectly calculating or acquiring the location of the mobile terminal.

The input unit 120 may be configured to provide an audio or video signal(or information) input to the mobile terminal or information input by auser to the mobile terminal. For the input of the audio information, themobile terminal 100 may include one or a plurality of cameras 121. Thecamera 121 processes a image frame, such as still picture or video,acquired by an image sensor in a video phone call or image capturingmode. The processed image frames may be displayed on the display module151. On the other hand, the plurality of cameras 121 disposed in themobile terminal 100 may be arranged in a matrix configuration. By use ofthe cameras 121 having the matrix configuration, a plurality of imageinformation having various angles or focal points may be input into themobile terminal 100. As another example, the cameras 121 may be locatedin a stereoscopic arrangement to acquire left and right images forimplementing a stereoscopic image.

The microphone 122 may process an external audio signal into electricaudio data. The processed audio data may be utilized in various mannersaccording to a function being executed in the mobile terminal 100 (or anapplication program being executed). On the other hand, the microphone122 may include assorted noise removing algorithms to remove noisegenerated in the course of receiving the external audio signal.

The user input unit 123 may receive information input by a user. Wheninformation is input through the user input unit 123, the controller 180may control an operation of the mobile terminal 100 to correspond to theinput information. The user input unit 123 may include one or more of amechanical input element (for example, a key, a button located on afront and/or rear surface or a side surface of the mobile terminal 100,a dome switch, a jog wheel, a jog switch, and the like), or atouch-sensitive input, among others. As one example, the touch-sensitiveinput may be a virtual key or a soft key, which is displayed on a touchscreen through software processing, or a touch key which is located onthe mobile terminal at a location that is other than the touch screen.On the other hand, the virtual key or the visual key may be displayed onthe touch screen in various shapes, for example, graphic, text, icon,video, or a combination thereof.

On the other hand, the sensing unit 140 may sense at least one ofinternal information of the mobile terminal, surrounding environmentinformation of the mobile terminal and user information, and generate asensing signal corresponding to it. The controller 180 may control anoperation of the mobile terminal 100 or execute data processing, afunction or an operation associated with an application programinstalled in the mobile terminal based on the sensing signal.Hereinafter, description will be given in more detail of representativesensors of various sensors which may be included in the sensing unit140.

First, a proximity sensor 141 refers to a sensor to sense presence orabsence of an object approaching to a surface to be sensed, or an objectdisposed near a surface to be sensed, by using an electromagnetic fieldor infrared rays without a mechanical contact. The proximity sensor 141may be arranged at an inner region of the mobile terminal covered by thetouch screen, or near the touch screen.

The proximity sensor 141 may include an optical transmission typephotoelectric sensor, a direct reflective type photoelectric sensor, amirror reflective type photoelectric sensor, a high-frequencyoscillation proximity sensor, a capacitance type proximity sensor, amagnetic type proximity sensor, an infrared rays proximity sensor, andso on. When the touch screen is implemented as a capacitance type, theproximity sensor 141 may sense proximity of a pointer to the touchscreen by changes of an electromagnetic field, which is responsive to anapproach of an object with conductivity. In this case, the touch screen(touch sensor) may also be categorized as a proximity sensor.

On the other hand, for the sake of brief explanation, a behavior inwhich the pointer is positioned to be proximate onto the touch screenwithout contact will be referred to as “proximity touch,” whereas abehavior in which the pointer substantially comes into contact with thetouch screen will be referred to as “contact touch.” For the positioncorresponding to the proximity touch of the pointer on the touch screen,such position will correspond to a position where the pointer facesperpendicular to the touch screen upon the proximity touch of thepointer. The proximity sensor 141 may sense proximity touch, andproximity touch patterns (e.g., distance, direction, speed, time,position, moving status, etc.). On the other hand, the controller 180may process data (or information) corresponding to the proximity touchesand the proximity touch patterns sensed by the proximity sensor 141, andoutput visual information corresponding to the process data on the touchscreen. In addition, the controller 180 may control the mobile terminal100 to execute different operations or process different data (orinformation) according to whether a touch with respect to the same pointon the touch screen is either a proximity touch or a contact touch.

A touch sensor can sense a touch applied to the touch screen, such asdisplay module 151, using any of a variety of touch methods. Examples ofsuch touch methods include a resistive type, a capacitive type, aninfrared type, and a magnetic field type, among others.

As one example, the touch sensor may be configured to convert changes ofpressure applied to a specific part of the display module 151 or acapacitance occurring from a specific part of the display module 151,into electric input signals. The touch sensor may be configured todetect a position, an area where a touch object applying a touch ontothe touch screen is touched on the touch sensor, a pressure at the timeof touch, a capacitance at the time of touch, and the like. Here, thetouch object body may be a finger, a touch pen or stylus pen, a pointer,or the like as an object through which a touch is applied to the touchsensor.

When a touch input is sensed by a touch sensor, corresponding signalsmay be transmitted to a touch controller. The touch controller mayprocess the received signals, and then transmit corresponding data tothe controller 180. Accordingly, the controller 180 may sense whichregion of the display module 151 has been touched. Here, the touchcontroller may be a component separate from the controller 180 or thecontroller 180 itself.

On the other hand, the controller 180 may execute a different control orthe same control according to a type of an object which touches thetouch screen (or a touch key provided in addition to the touch screen).Whether to execute the different control or the same control accordingto the object which gives a touch input may be decided based on acurrent operating state of the mobile terminal 100 or a currentlyexecuted application program.

Meanwhile, the touch sensor and the proximity sensor may be executedindividually or in combination, to sense various types of touches, suchas a short (or tap) touch, a long touch, a multi-touch, a drag touch, aflick touch, a pinch-in touch, a pinch-out touch, a swype touch, ahovering touch, and the like.

An ultrasonic sensor may be configured to recognize position informationrelating to a sensing object by using ultrasonic waves. Meanwhile, thecontroller 180 may calculate the location of a wave generating sourcethrough information sensed from the optical sensor and the plurality ofultrasonic sensors. Since light is much faster than ultrasonic waves, atime for which the light reaches the optical sensor may be much shorterthan a time for which the ultrasonic wave reaches the ultrasonic sensor.The position of the wave generation source may be calculated using thefact. In more detail, the position of the wave generation source may becalculated by using a time difference from the time that the ultrasonicwave reaches based on the light as a reference signal.

The camera 121 constructing the input unit 120 may be a type of camerasensor. The camera sensor may include at least one of a photo sensor (orimage sensor) and a laser sensor.

The camera 121 and the laser sensor may be combined with each other todetect a touch of the sensing object with respect to a 3D stereoscopicimage. The photo sensor may be laminated on the display device. Thephoto sensor may be configured to scan a movement of the sensing objectin proximity to the touch screen. In more detail, the photo sensor mayinclude photo diodes and transistors at rows and columns to scan contentplaced on the photo sensor by using an electrical signal which changesaccording to the quantity of applied light. Namely, the photo sensor maycalculate the coordinates of the sensing object according to variationof light to thus obtain position information of the sensing object.

The display module 151 may display (output) information processed in themobile terminal 100. For example, the display module 151 may displayexecution screen information of an application program driven in themobile terminal 100 or user interface (UI) and graphic user interface(GUI) information in response to the execution screen information.

Furthermore, the display module 151 may also be implemented as astereoscopic display module for displaying stereoscopic images.

The stereoscopic display module may employ a stereoscopic display schemesuch as stereoscopic scheme (a glass scheme), an auto-stereoscopicscheme (glassless scheme), a projection scheme (holographic scheme), orthe like.

The audio output unit 152 may output audio data received from thewireless communication unit 110 or stored in the memory 160 in a callsignal reception mode, a call mode, a record mode, a voice recognitionmode, a broadcast reception mode, and the like. Also, the audio outputmodule 152 may also provide audible output signals associated with aparticular function (e.g., a call signal reception sound, a messagereception sound, etc.) carried out by the mobile terminal 100. The audiooutput module 152 may include a receiver, a speaker, a buzzer or thelike.

A haptic module 153 may generate various tactile effects the that usermay feel. A typical example of the haptic effect that is generated bythe haptic module 153 is vibration. Strength, pattern and the like ofthe vibration generated by the haptic module 153 may be controllable bya user selection or setting of the controller. For example, the hapticmodule 153 may output different vibrations in a combined or sequentialmanner.

Besides vibration, the haptic module 153 can generate various othertactile effects, including an effect by stimulation such as a pinarrangement vertically moving to contact skin, a spray force or suctionforce of air through a jet orifice or a suction opening, a touch to theskin, a contact of an electrode, electrostatic force, an effect byreproducing the sense of cold and warmth using an element that canabsorb or generate heat, and the like.

The haptic module 153 can also be implemented to allow the user to feela tactile effect through a muscle sensation such as the user's fingersor arm, as well as transferring the tactile effect through directcontact. Two or more haptic modules 153 may be provided according to theparticular configuration of the mobile terminal 100.

An optical output module 154 may output a signal for indicating an eventgeneration using light of a light source. Examples of events generatedin the mobile terminal 100 may include a message reception, a callsignal reception, a missed call, an alarm, a schedule notice, an emailreception, an information reception through an application, and thelike.

A signal output by the optical output module 154 may be implemented insuch a manner that the mobile terminal emits monochromatic light orlight with a plurality of colors. The signal output may be terminated asthe mobile terminal senses that a user has checked the generated event,for example.

The interface unit 160 serves as an interface for external devices to beconnected with the mobile terminal 100. For example, the interface unit160 can receive data transmitted from an external device, receive powerto transfer to elements and components within the mobile terminal 100,or transmit internal data of the mobile terminal 100 to such externaldevice. The interface unit 160 may include wired or wireless headsetports, external power supply ports, wired or wireless data ports, memorycard ports, ports for connecting a device having an identificationmodule, audio input/output (I/O) ports, video I/O ports, earphone ports,or the like.

The identification module may be a chip that stores various informationfor authenticating authority of using the mobile terminal 100 and mayinclude a user identity module (UIM), a subscriber identity module(SIM), a universal subscriber identity module (USIM), and the like. Adevice having the identification module (hereinafter, an “identificationdevice”) may be fabricated in the form of a smart card. Accordingly, theidentifying device may be connected with the terminal 100 via theinterface unit 160.

Furthermore, when the mobile terminal 100 is connected with an externalcradle, the interface unit 160 may serve as a passage to allow powerfrom the cradle to be supplied to the mobile terminal 100 therethroughor may serve as a passage to allow various command signals input by theuser from the cradle to be transferred to the mobile terminaltherethrough. Such various command signals or power inputted from thecradle may operate as signals for recognizing that the mobile terminal100 has accurately been mounted to the cradle.

The memory 170 can store programs to support operations of thecontroller 180 and store input/output data (for example, phonebook,messages, still images, videos, etc.). The memory 170 may store datarelated to various patterns of vibrations and audio which are output inresponse to touch inputs on the touch screen.

The memory 170 may include at least one type of storage medium includinga Flash memory, a hard disk, a multimedia card micro type, a card-typememory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), aStatic Random Access Memory (SRAM), a Read-Only Memory (ROM), anElectrically Erasable Programmable Read-Only Memory (EEPROM), aProgrammable Read-Only memory (PROM), a magnetic memory, a magneticdisk, and an optical disk. Also, the mobile terminal 100 may be operatedin relation to a web storage device that performs the storage functionof the memory 170 over the Internet.

As aforementioned, the controller 180 may typically control the generaloperations of the mobile terminal 100. For example, the controller 180may execute or release a locked state for restricting a user frominputting a control command with respect to applications when a state ofthe mobile terminal meets a preset condition.

Furthermore, the controller 180 may also perform controlling andprocessing associated with voice calls, data communications, videocalls, and the like, or perform pattern recognition processing torecognize a handwriting input or a picture drawing input performed onthe touch screen as characters or images, respectively. In addition, thecontroller 180 may control one or combination of those components inorder to implement various exemplary embodiment disclosed herein on themobile terminal 100.

The power supply unit 190 may receive external power or internal powerand supply appropriate power required for operating respective elementsand components included in the mobile terminal 100 under the control ofthe controller 180. The power supply unit 190 may include a battery,which is typically rechargeable or be detachably coupled to the terminalbody for charging.

Furthermore, the power supply unit 190 may include a connection port.The connection port may be configured as one example of the interfaceunit 160 to which an external (re)charger for supplying power torecharge the battery is electrically connected.

As another example, the power supply unit 190 may be configured torecharge the battery in a wireless manner without use of the connectionport. Here, the power supply unit 190 may receive power, transferredfrom an external wireless power transmitter, using at least one of aninductive coupling method which is based on magnetic induction or amagnetic resonance coupling method which is based on electromagneticresonance.

Various embodiments described herein may be implemented in acomputer-readable or its similar medium using, for example, software,hardware, or any combination thereof.

Referring to FIGS. 1B and 10 , the mobile terminal 100 disclosed hereinmay be provided with a bar-type terminal body. However, the presentdisclosure may not be necessarily limited to this, and may be alsoapplicable to various structures such as a watch type, a clip type, aglasses type, a folder type in which two or more bodies are coupled toeach other in a relatively movable manner, a slide type, a swing type, aswivel type, and the like. The present disclosure relates to a specifictype of mobile terminal, but the description of a specific type ofmobile terminal may be also applicable to another type of mobileterminal in general.

Here, the terminal body may be understood as a conception whichindicates the mobile terminal 100 as at least one assembly.

The mobile terminal 100 may include a case (for example, a frame, ahousing, a cover, etc.) constituting the appearance thereof. In thisembodiment, the case may be divided into a front case 101 and a rearcase 102. Various electronic components may be incorporated into a spaceformed between the front case 101 and the rear case 102. At least onemiddle case may be additionally disposed between the front case 101 andthe rear case 102

A display module 151 may be disposed on a front surface of the terminalbody to output information. As illustrated, a window 151 a of thedisplay module 151 may be mounted to the front case 101 so as to formthe front surface of the terminal body together with the front case 101.

In some cases, electronic components may also be mounted to the rearcase 102. Examples of those electronic components mounted to the rearcase 102 may include a detachable battery, an identification module, amemory card and the like. Here, a rear cover 103 for covering theelectronic components mounted may be detachably coupled to the rear case102. Therefore, when the rear cover 103 is detached from the rear case102, the electronic components mounted to the rear case 102 may beexternally exposed.

As illustrated, when the rear cover 103 is coupled to the rear case 102,a side surface of the rear case 102 may be partially exposed. In somecases, upon the coupling, the rear case 102 may also be completelyshielded by the rear cover 103. On the other hand, the rear cover 103may include an opening for externally exposing a camera 121 b or anaudio output module 152 b.

The cases 101, 102, 103 may be formed by injection-molding syntheticresin or may be formed of a metal, for example, stainless steel (STS),aluminum (Al), titanium (Ti), or the like.

Unlike the example which the plurality of cases form an inner space foraccommodating such various components, the mobile terminal 100 may beconfigured such that one case forms the inner space. In this case, themobile terminal 100 with a unibody in which synthetic resin or metal isextended from a lateral surface to a rear surface may be implemented.

On the other hand, the mobile terminal 100 may include a waterproofingunit (not shown) for preventing an introduction of water into theterminal body. For example, the waterproof portion may include awaterproof member provided between the window 151 a and the front case101, between the front case 101 and the rear case 102, or between therear case 102 and the back cover 103, to hermetically seal an innerspace when those cases are coupled to each other.

The mobile terminal 100 may include a display module 151, first andsecond audio output modules 152 a and 152 b, a proximity sensor 141, anillumination sensor 152, an optical output module 154, first and secondcameras 121 a and 121 b, first and second manipulation units 123 a and123 b, a microphone 122, an interface unit 160 and the like.

Hereinafter, description will be given of an exemplary mobile terminal100 that the display module 151, the first audio output module 152 a,the proximity sensor 141, the illumination sensor 142, the opticaloutput module 154, the first camera 121 a and the first manipulationunit 123 a are disposed on the front surface of the terminal body, thesecond manipulation unit 123 b, the microphone 122 and the interfaceunit 160 are disposed on a side surface of the terminal body, and thesecond audio output module 152 b and the second camera 121 b aredisposed on a rear surface of the terminal body, with reference to FIGS.1B and 10 .

However, the foregoing configuration may not be necessarily limited tothe arrangement. The foregoing configuration may be excluded,substituted or disposed on another surface if necessary. For example,the first manipulation unit 123 a may not be disposed on the frontsurface of the terminal body, and the second audio output module 152 bmay be disposed on the side surface other than the rear surface of theterminal body.

The display module 151 displays (outputs) information processed by themobile terminal 100. For example, the display module 151 may displayexecution screen information of an application program driven in themobile terminal 100 or user interface (UI) and graphic user interface(GUI) information in response to the execution screen information.

The display module 151 may include at least one of a liquid crystaldisplay (LCD), a thin film transistor-LCD (TFT-LCD), an organiclight-emitting diode (OLED), a flexible display, a three-dimensional(3D) display and an e-ink display.

The display module 151 may be implemented in two or more in numberaccording to a configured aspect of the mobile terminal 100. Forinstance, a plurality of the display modules 151 may be arranged on onesurface to be spaced apart from or integrated with each other, or may bearranged on different surfaces.

The display module 151 may include a touch sensor which senses a touchonto the display module so as to receive a control command in a touchingmanner. When a touch is input to the display module 151, the touchsensor may be configured to sense this touch and the controller 180 maygenerate a control command corresponding to the touch. The content whichis input in the touching manner may be a text or numerical value, or amenu item which can be indicated or designated in various modes.

The touch sensor may be configured in a form of a film having a touchpattern, disposed between the window 151 a and a display on a rearsurface of the window 151 a, or a metal wire which is patterned directlyon the rear surface of the window 151 a. Alternatively, the touch sensormay be integrally formed with the display. For example, the touch sensormay be disposed on a substrate of the display or within the display.

The display module 151 may form a touch screen together with the touchsensor. Here, the touch screen may serve as the user input unit 123 (seeFIG. 1A). Therefore, the touch screen may replace at least some of thefunctions of the first manipulation unit 123 a.

The first audio output module 152 a may be implemented in the form of areceiver for transferring voice sounds to the user's ear or a loudspeaker for outputting various alarm sounds or multimedia reproductionsounds.

The window 151 a of the display module 151 may include a sound hole foremitting sounds generated from the first audio output module 152 a.Here, the present disclosure may not be limited to this. It may also beconfigured such that the sounds are released along an assembly gapbetween the structural bodies (for example, a gap between the window 151a and the front case 101). In this case, a hole independently formed tooutput audio sounds may not be seen or is otherwise hidden in terms ofappearance, thereby further simplifying the appearance and manufacturingof the mobile terminal 100.

The optical output module 154 may output light for indicating an eventgeneration. Examples of the event generated in the mobile terminal 100may include a message reception, a call signal reception, a missed call,an alarm, a schedule notice, an email reception, information receptionthrough an application, and the like. When a user's event check issensed, the controller 180 may control the optical output unit 154 toend the output of light.

The first camera 121 a may process video frames such as still or movingimages acquired by the image sensor in a video call mode or a capturemode. The processed video frames may be displayed on the display module151 or stored in the memory 170.

The first and second manipulation units 123 a and 123 b are examples ofthe user input unit 123, which may be manipulated by a user to input acommand for controlling the operation of the mobile terminal 100. Thefirst and second manipulation units 123 a and 123 b may employ anymethod if it is a tactile manner allowing the user to performmanipulation with a tactile feeling such as touch, push, scroll or thelike. In addition, the first and second manipulation units 123 a and 123b may also employ a method of allowing the user to perform manipulationwithout a tactile feeling through a proximity touch, a hovering touch,or the like.

The drawings are illustrated on the basis that the first manipulationunit 123 a is a touch key, but the present disclosure may not benecessarily limited to this. For example, the first manipulation unit123 a may be configured with a mechanical key, or a combination of atouch key and a push key.

The content received by the first and second manipulation units 123 aand 123 b may be set in various ways. For example, the firstmanipulation unit 123 a may be used by the user to input a command suchas menu, home key, cancel, search, or the like, and the secondmanipulation unit 123 b may be used by the user to input a command, suchas controlling a volume level being output from the first or secondaudio output module 152 a or 152 b, switching into a touch recognitionmode of the display module 151, or the like.

On the other hand, as another example of the user input unit 123, a rearinput unit (not shown) may be disposed on the rear surface of theterminal body. The rear input unit may be manipulated by a user to inputa command for controlling an operation of the mobile terminal 100. Thecontent input may be set in various ways. For example, the rear inputunit may be used by the user to input a command, such as power on/off,start, end, scroll or the like, controlling a volume level being outputfrom the first or second audio output module 152 a or 152 b, switchinginto a touch recognition mode of the display module 151, or the like.The rear input unit may be implemented into a form allowing a touchinput, a push input or a combination thereof.

The rear input unit may be disposed to overlap with the display module151 of the front surface in a thickness direction of the terminal body.As one example, the rear input unit may be disposed on an upper endportion of the rear surface of the terminal body such that a user caneasily manipulate it using a forefinger when the user grabs the terminalbody with one hand. However, the present disclosure may not be limitedto this, and the position of the rear input unit may be changeable.

When the rear input unit is disposed on the rear surface of the terminalbody, a new user interface may be implemented using the rear input unit.Also, the aforementioned touch screen or the rear input unit maysubstitute for at least part of functions of the first manipulation unit123 a located on the front surface of the terminal body. Accordingly,when the first manipulation unit 123 a is not disposed on the frontsurface of the terminal body, the display module 151 may be implementedto have a larger screen.

On the other hand, the mobile terminal 100 may include a fingerprintrecognition sensor for recognizing a users fingerprint, and thecontroller 180 may use fingerprint information sensed through the fingerrecognition sensor as an authentication means. The finger recognitionsensor may be integrated into the display module 151 or the user inputunit 123.

The microphone 122 may be formed to receive the user's voice, othersounds, and the like. The microphone 122 may be provided at a pluralityof places, and configured to receive stereo sounds.

The interface unit 160 may serve as a path allowing the mobile terminal100 to exchange data with external devices. For example, the interfaceunit 160 may be at least one of a connection terminal for connecting toanother device (for example, an earphone, an external speaker, or thelike), a port for near field communication (for example, an InfraredData Association (IrDA) port, a Bluetooth port, a wireless LAN port, andthe like), or a power supply terminal for supplying power to the mobileterminal 100. The interface unit 160 may be implemented in the form of asocket for accommodating an external card, such as SubscriberIdentification Module (SIM), User Identity Module (UIM), or a memorycard for information storage.

The second camera 121 b may be further mounted to the rear surface ofthe terminal body. The second camera 121 b may have an image capturingdirection, which is substantially opposite to the direction of the firstcamera unit 121 a.

The second camera 121 b may include a plurality of lenses arranged alongat least one line. The plurality of lenses may also be arranged in amatrix configuration. The cameras may be referred to as an ‘arraycamera.’ When the second camera 121 b is implemented as the arraycamera, images may be captured in various manners using the plurality oflenses and images with better qualities may be obtained.

A flash 124 may be disposed adjacent to the second camera 121 b. When animage of a subject is captured with the camera 121 b, the flash 124 mayilluminate the subject.

The second audio output module 152 b may further be disposed on theterminal body. The second audio output module 152 b may implement astereo function in conjunction with the first audio output module 152 a,and may be also used to implement a speakerphone mode during a call.

At least one antenna for wireless communication may be disposed on theterminal body. The antenna may be installed in the terminal body orformed on the case. For example, an antenna which configures a part ofthe broadcast receiving module 111 (see FIG. 1A) may be retractable intothe terminal body. Alternatively, an antenna may be formed in a form offilm to be attached onto an inner surface of the rear cover 103 or acase including a conductive material may serve as an antenna.

A power supply unit 190 (refer to FIG. 1A) for supplying power to themobile terminal 100 may be disposed on the terminal body. The powersupply unit 190 may include a batter 191 which is mounted in theterminal body or detachably coupled to an outside of the terminal body.

The battery 191 may receive power via a power source cable connected tothe interface unit 160. Furthermore, the battery 191 may be(re)chargeable in a wireless manner using a wireless charger. Thewireless charging may be implemented by magnetic induction orelectromagnetic resonance.

On the other hand, the drawing illustrates that the back cover 103 iscoupled to the rear case 102 for shielding the battery 191, so as toprevent separation of the battery 191 and protect the battery 191 froman external impact or foreign materials. When the battery 191 isdetachable from the terminal body, the rear case 103 may be detachablycoupled to the rear case 102.

An accessory for protecting an appearance or assisting or extending thefunctions of the mobile terminal 100 can also be provided on the mobileterminal 100. As one example of an accessory, a cover or pouch forcovering or accommodating at least one surface of the mobile terminal100 may be provided. The cover or pouch may link with the display module151 to extend the function of the mobile terminal 100. Another exampleof the accessory may be a touch pen for assisting or extending a touchinput onto a touch screen.

The mobile terminal 100 according to an embodiment of the presentdisclosure described above may include the camera 121 for capturing animage, and a preview image for capturing is displayed on the displaymodule 151 when a camera application is executed. Then, when it issensed that the camera operation has entered an information search mode,the controller 180 may provide a plurality of search options for asubject.

Then, when any one of the presented plurality of search options issensed, and then a touch input is applied to the preview image, thecontroller 180 may execute a capture command for capturing a subjectincluding an object to which the selected search option is applied.Then, the controller 180 may display an image of the captured subject onthe display module 151 along with an application screen of the selectedsearch option.

Accordingly, the user may easily and quickly acquire a desiredinformation search result by simply capturing a subject without havingto directly perform information search related to an object in reality.Furthermore, a desired option may be accurately specified among aplurality of search options prior to image analysis on a subject,thereby eliminating errors due to automatic selection of a search objectand a search option.

Hereinafter, the information search mode described in the presentdisclosure denotes an operation mode capable of performing variousinformation search on a current screen without executing an additionalapplication for information included in all or a specific part of thesubject.

Furthermore, the information search mode may denote an operation modecapable of performing various information search functions on a currentscreen without loading a pre-stored image and executing an additionalapplication. In other words, the present disclosure may be applied notonly to a preview image but also similarly to an image previously storedin the terminal.

In addition, the plurality of search options for the subject, as aplurality of search items applicable to information included in thesubject, may all include various search options provided by a typicalweb search page, such as image search, shopping search, web search,dictionary search, translation search, news search, map search, contentsearch, and book search, and the like.

On the other hand, the plurality of search options for the subject maybe the same regardless of the subject or image, and for another example,the subject or image may be different depending on the location or timeat which the subject or image is captured.

Hereinafter, with reference to FIGS. 2A, 2B, 2C and 2D, a specificoperation associated with the execution of an information search mode ofa subject according to the present disclosure will be described.

First, referring to FIG. 2A, a preview image 201 displayed according tothe execution of a camera application in the mobile terminal 100according to the present disclosure is shown.

At this time, a home touch button including a back icon is provided atthe bottom of the display module 151 as a default. Furthermore, afrequently used camera setting menu is displayed on the top of thedisplay module 151 as a default. In addition, in order to secure thefingering distance, a function icon 211 for entering an informationsearch mode according to the present disclosure may be presented bybeing included in a USP menu bar 210 at the top of a capture button 250,that is, directly under the preview image 201. However, this is merelyan example, and the structure in which the USP menu bar 210 is disposedon the side or the top of the display module 151 may also be possible.

Subsequently, referring to FIG. 2B, when a touch is applied to thefunction icon 211 for entering the information search mode according tothe present disclosure while a preview image 202 is displayed, themobile terminal enters an operation mode for executing a search functionon information included in the preview image 202 while maintaining thepreview image 202.

Accordingly, as illustrated in FIG. 2C, a user interface 230 including aplurality of search options for a subject included in the preview image202 may be displayed at a location where the capture button 250 waslocated.

Here, the plurality of search options may be representative searchoptions presented as defaults regardless of a subject viewed through thecamera 121. For example, as shown in FIG. 2C, a shopping search option,an image search option, a text scan option, and a QR scan option may beprovided as default menu icons in the information search mode in theuser interface 230.

The user may select a desired specific search option, for example, ashopping search option 231 from various search options presented in theuser interface 230 through a touch input.

However, in one example, the selection of the specific search option maybe performed without any user input. For example, the shopping searchoption 231 or another specific search option may be automaticallyselected as an initial search option. In this case, when theautomatically selected search option matches the desired search option,the user may skip the selection process of the search option to performthe next step.

Then, guide information 220 for inducing the selection of an object towhich the selected shopping search option 231 is applied is displayed inan area, for example, at the bottom of the preview image 202. In otherwords, a specific search object is specified after the search option isspecified first. In general, when searching through a web page, a searchis executed by entering a search keyword, and then a specific searchoption is selected to acquire a specific search result. To this end, atleast initially, the range of DB used for searching is applied to alloptions such as image, shopping, document, dictionary, video, news, andthe like. Even if the user only wanted to perform an image searchclearly from the beginning, he or she had to check the entire searchresults and then select the image search option.

However, in the present disclosure, it may be implemented to firstselect a specific search option such as shopping or image, and thenselect a specific search object, that is, a specific object within thepreview image 202, thereby allowing the user to more directly reachdesired results, and minimizing resource loss rather than providing afull range of search results.

In addition, without entering a search term, the search object may belimited by simply pointing an object in real through the camera, and asearch object may be more reliably specified only by touching a specificobject on the preview image 202 according to the guide information 220illustrated in FIG. 2C.

For example, when a single touch is applied to a specific object in FIG.2C, any one object corresponding to a touch point to which a singletouch is applied is specified among a plurality of divisible objects inthe preview image 202. However, the single touch is an example, and mayalso be replaced by another type of touch such as a double touch or amethod of drawing a line along an outline of the object.

Subsequently, in FIG. 2C, when a touch input is applied to a “cupportion” of a specific object in the preview image 202, for example, “acake in a cup”, image analysis for the “cup portion” in the previewimage 202 is carried out. Therefore, in the present disclosure, imageanalysis does not need to be performed until a search option isspecified and an object to be searched is selected, thereby reducing aload due to unnecessary image analysis. In addition, image analysis forsearch is also performed only on the selected portion rather than theentire subject.

In one example, when selecting an object in the preview image 202, apredetermined graphic object that visually recognizes the selection maybe displayed. Furthermore, in another example, a guide image may bedisplayed along an outline of the selected object, and implemented torecognize that the relevant object is finally selected through afeedback input to the guide image.

On the other hand, a touch on the preview image 202 is also performed asan input of a capture signal. In other words, using a touch signal as aninput signal, the selection of an object and capturing on a subject arecarried out at the same time.

When the object is a portion (e.g., a cup portion) of the subject (e.g.,a cake in a cup) as shown in FIG. 2C, an image of the captured subjectand an object to which a shopping search option is applied may bedifferent.

Subsequent to capturing a subject and selecting an object, when imageanalysis for the selected object is completed, a shopping search isexecuted based on information (e.g., a shape of the cup, a material ofthe cup, etc.) corresponding to the analysis result. While the shoppingsearch is being executed, a loading screen may be displayed at thebottom of the preview image 202.

When a predetermined time period has elapsed, an application screen 260of the shopping search for the selected object is displayed along withthe image 203 of the captured subject.

The application screen 260 may be in the form of a bottom sheetwithdrawn from the bottom of the display module 151, as shown in FIG.2D. Accordingly, the bottom sheet may be drawn in while moving back tothe bottom of the display module 151 or drawn out again while movingupward according to a flicking touch input applied to the applicationscreen 260 in a vertical direction. Furthermore, the application screen260 is displayed by overlapping with a portion of the image 203 of thecaptured subject according to the drawn-out.

In addition, a plurality of items corresponding to a result of theshopping search are displayed on the application screen 260, and imageand product information (for example, a seller, a product price, etc.)is included in each item. At this time, when a flicking touch input isapplied to the application screen 260 in a vertical or horizontaldirection, the items may be scrolled to display some other items or toswitch to another page including other items.

When the user selects any one item from the application screen 260, aweb page shortcut linked to the selected item is executed. Accordingly,the captured subject image 203 is immediately switched to a specific webpage screen.

As described above, in the present disclosure, a search object may bespecified by simply pointing the camera, and a specific search optionmay be first selected and then an object to be searched may be selectedat the same time as capturing, and thus there is no need to additionallyexecute an application for search, and a user's desired search optionmay be immediately applied to a specific object.

Hereinafter, FIG. 3 is a representative flowchart for explaining aspecific operation according to the execution of a subject informationsearch mode in a mobile terminal associated with the present disclosure.

First, a preview image for the shooting of a camera is displayed on thedisplay module 151 of the mobile terminal 100 (S10).

While the preview image is displayed, the controller 180 may detect theentry of a subject into the information search mode (S20). Theinformation search mode may be performed through an input signal by auser or a selection signal for a request to prevent unnecessary programsfrom continuously running in the background.

Next, in the information search mode, the selection of one of aplurality of search options for the subject is sensed (S30).

The selection of any one of the plurality of search options may be madethrough a user input. For example, when a touch input is applied to anyone of the plurality of search option items presented through a userinterface (UI), it may be recognized that the relevant item is selected.As described above, a search range is clearly specified every timethrough the user input, and different search options may be selected forthe same subject, for example.

In another example, any one of the plurality of search options may beselected based on sensor values sensed through various sensors providedin the mobile terminal 100 and location information or time informationcollected by the mobile terminal 100. In this case, an image analysisresult of the subject may be additionally used.

Furthermore, in still another example, the plurality of search optionsthemselves may be presented differently according to the subject or theuser's preference. In this case, the selection of any one of theplurality of search options may affect the type and arrangement of theplurality of search options in the future. In other words, using aselection from the plurality of search options as a feedback inputvalue, the plurality of search options presented next may be controlled.

In addition, the plurality of search options may be scrolled accordingto an additional input, for example, a horizontal or vertical flickinginput, to provide more search options.

Any one of the search options is specified as described above, and thena subject including an object to which the search option selected instep S30 is applied is captured based on a touch input applied to thepreview image (S40).

To this end, the controller 180 may transmit a capture command to thecamera 121 at a time point when a touch input is applied to the previewimage while at the same time performing image analysis on an object towhich the selected search option is applied.

Then, an image of the captured subject is displayed on the displaymodule 151 along with an object search option application screen (S50).At this time, the captured subject and the object to which the searchoption is applied may be different from each other, or either one ofthem may have a partial relationship to the other one.

When any one search result is selected on the search option applicationscreen, it may be immediately moved to its related screen.Alternatively, after the search option application screen is displayed,a search option or an object to be searched may be changed and searchedagain through a back button.

Hereinafter, FIGS. 4A, 4B, 4C, 4D, 4E and 4F illustrate examples ofquickly specifying or changing a desired object to execute aninformation search function when a plurality of objects are included inan object to be captured.

First, referring to FIG. 4A, when a plurality of objects, for example, akettle and a lamp, are included in a preview image 401 to be captured,the selection range of an object varies according to informationdisplayed at a touch point of a touch input applied to the preview image401.

Specifically, when a shopping search option is selected, and then atouch input (T1) is applied to a background region of the preview image401, the shopping search option is applied to all objects included inthe preview image 401. To this end, the controller 180 may distinguishand recognize whether the information displayed at a point where thetouch input is applied is a background image or a specific object image.In other words, when a location applied to the touch point belongs to abackground image, the background image is not recognized as an object,but is processed that all the objects are selected.

In this case, as shown in FIG. 4B, a shopping search is executed for allof a plurality of objects (e.g., kettle, lamp). At this time, guideimages may be displayed on the objects 411, 412 selected from thecaptured image 402.

At this time, not only an item 261 of the shopping search result for thefirst object 411 and an item 262 of the shopping search result for thesecond object 412 but also an item 263 of the shopping search resultincluding all the plurality of objects may be provided on theapplication screen 260.

When a back execution command is input while the application screen 260of the selected search option is displayed, it may be switched to ascreen for changing an object to which the selected search option is tobe applied.

Accordingly, as illustrated in FIG. 4C, the user interface 230 displayedwith a plurality of search options is displayed again along with thepreview image 401. Here, the user may apply a touch input to anotherobject included in the preview image 401 to change an object to whichthe search option is applied. Alternatively, another search option mayalso be selected from the user interface 230. Alternately, it may bepossible to change both the search option and object.

When a touch input (T2) is applied to a “lamp” 412 in the preview image401 of FIG. 4C, the shopping search object is changed from “lamp,kettle, lamp +kettle” to “lamp”. At this time, a guide image 10 achanged to visually recognize the changed object is displayed.

Then, when a predetermined time has elapsed, as illustrated in FIG. 4D,only items 264, 262, 265 of the shopping search result related to the“lamp” are displayed on the application screen 260 of the search option.

To this end, when a touch input is applied to only one object in apreview image including a plurality of objects, the controller 180limits the application of the selected search option to the remainingobjects distinguished from it. For example, in FIG. 4D, it may be seenthat the shopping search option is not applied to the “kettle” 411.

Then, when it is desired to change the search object from the “lamp” tothe “kettle” 411 again, it may be achieved by selecting a button forgenerating a back execution command, then applying a touch input (T3) tothe “kettle” in the preview image 401 of FIG. 4E.

As described above, according to a change of the search object, thelocation of the guide image 10 b is also changed to be different fromthe previous one. As a result, as illustrated in FIG. 4F, only items261, 266, 267 of the shopping search result related to the “kettle” aredisplayed on the application screen 260 of the search option.

As described above, in the present disclosure, a desired search resultmay be acquired by selecting a desired search option from a previewimage including a plurality of objects, and then selectively specifyingany one object from the beginning or including all of the objects.Moreover, it may be possible to quickly change a target object or asearch option itself after checking the search result.

Hereinafter, FIGS. 5A, 5B, 5C and 5D are exemplary views for explaininga specific operation of executing an image search function on a subject.

First, referring to FIG. 5A, when an image search option 232 is selectedin the information search mode, guide information 220 for inducing theselection of an object to be captured to search for an image isdisplayed at the bottom of the preview image 501.

In response to the guide information 220, when a touch input is appliedto an object in the preview image 501, for example, “a cake in a cup”,an image search for the relevant object is performed at the same time ascapturing. At this time, the captured image is stored in the memory 160of the mobile terminal, for example.

Then, as illustrated in FIG. 5C, the captured image 502 is displayedfirst, and a loading screen 240 is provided at the bottom while imageanalysis and image search on the object are performed. At this time, thecaptured image 502 may be displayed with an outline image indicating aportion where object recognition is actually performed for the imagesearch.

Next, as shown in FIG. 5D, an application screen 560 that provides animage search result related to “a cake in a cup” is displayed whileoverlapping with at least a portion of the captured image 502. The usermay select one of a plurality of images provided on the applicationscreen 560 to view the relevant image in an enlarged manner or directlyaccess a page related to the relevant image.

As described above, in the present disclosure, the function may beapplied in an extended manner to an area of search beyond capturethrough the preview image of the camera 121, and in particular, acustomized information search may be possible by selecting a desiredsearch option first, and then using information possessed by the previewimage at a faster speed.

Hereinafter, specific examples related to a case where informationpossessed by the preview image includes text will be described.

FIGS. 6, 7A, 7B, 7C, 7D, 8A, 8B, 8C, 8D, 9A, 9B, 9C, 9D, 10A and 10Billustrate operations of extracting text included in an image of asubject to provide an additional function thereto in a mobile terminalassociated with the present disclosure.

First, FIG. 6 is a flowchart for explaining an operation of executing aninformation search mode of a subject according to the present disclosureto extract text included in a preview image and provide an additionalfunction related thereto.

In FIG. 6 , first, a preview image for the shooting of a camera isdisplayed on the display module 151 of the mobile terminal 100 (S601).While the preview image is displayed, the entry of a subject into theinformation search mode may be detected (S602). In the informationsearch mode, the selection of a text search option among a plurality ofsearch options for a subject is detected (S603).

At this time, the selection of the text search option may be madethrough an explicit user input or initial setting. For example, a textsearch option may be selected by applying a touch input to a text searchoption among a plurality of search options presented on the displaymodule 151. Alternatively, when the initial setting is set as a textsearch option from the beginning or the search option that has beenlately carried out is a text search option, it may be recognized thatthe text search option is selected exceptionally without explicit user'sselection.

In another example, when text information is detected in a subjectimage, guide information for inducing the selection of a text searchoption among the plurality of search options, for example, ahighlighting effect output or the like, may be displayed. However, evenat this time, the final selection of the text search option may be madeby the user.

Subsequently, text information to which the selected text search optionis to be applied is extracted based on a touch input applied to thepreview image (S604).

At this time, only part of text included in the preview image may beselected. This may be achieved through a process similar to selectingany one from a preview image including a plurality of objects describedabove.

When text information is extracted, the preview image is replaced withan extraction screen displayed with the extracted text information. Inother words, the extracted text information is reconstructed anddisplayed in a predetermined font and font size, other than the previewimage itself. Furthermore, the plurality of search options that havebeen displayed are switched to a function icon related to the extractionscreen and displayed (S605).

Here, the function related to the extraction screen is a functionrelated to the processing of text information included in the extractionscreen, and does not simply denote storage, transmission, and editing.In other words, a storage location and a storage method, such as savinga contact, saving a memo, saving a PDF, or the like, may be providedaccording to the attributes of text information included in theextraction screen and the user's usage habits.

The user may quickly process text information included in the extractionscreen in a desired manner by selecting any one of the displayedfunction icons.

FIGS. 7A through 7D show examples of providing a result of applying atranslation function to text information extracted from a preview image.

In FIG. 7A, when a document including text information is pointed by thecamera, and accordingly, the selection of a text search option 233 amongthe plurality of search options is detected while the preview image 701is displayed, guide information 220 for inducing the selection a textarea to which the text search option is to be applied is displayed.

Next, as illustrated in FIG. 7B, when a touch input (T4) is applied to aspecific text area in the displayed preview image 701, text displayed inthe relevant area is extracted to recognize the content. At this time,in order to increase the text extraction and recognition rate, aflashlight mode 730 may be automatically activated in consideration of acurrent battery level.

When a predetermined time has elapsed, as illustrated in FIG. 7C, anextraction screen 702 configured with text information extracted andrecognized from the selected text area is displayed by replacing thepreview image 701. In addition, in FIG. 7B, a plurality of searchoptions displayed at the bottom of the display module 151 are switchedto function icons 730 related to text information included in theextraction screen 702.

At this time, the function icons 730 may be determined differentlyaccording to the attributes and contents of text information included inthe extraction screen 702. For example, in the case of a “document”including a “foreign language” as shown in FIG. 7B, a PDF format storageicon 733 and a translation function icon 734 may be provided as functionicons 730 related to text information.

In addition, in one example, information to which an additional functionis to be applied may be further restricted by controlling thesize/location of a guide image displayed on a boundary line of theextraction screen 702 through a drag touch input, or selecting aspecific word, line, or a plurality of words/lines included in theextraction screen 702. In this case, the function icons 730 that havealready been displayed may vary according to the attributes and contentsof the restricted information.

Meanwhile, in FIG. 7C, when the translation icon 734 is selected, atranslation function corresponding to the selected translation icon 734is executed on text information of the extraction screen 702.

Accordingly, as illustrated in FIG. 7D, a translation screen 703translated from “English” included in the extraction screen 702 to“Korean” is drawn out from the bottom of the display module 151 towardthe top as an application screen 260 in the form of a bottom sheet. Thetranslation screen 703 is provided with a translation result along witha user interface capable of selecting a translation language.

In addition, in one example, when a touch input is applied to anapplication screen of a function corresponding to a function icon, thatis, the translation screen 703, information displayed on the applicationscreen may be saved along with the extraction screen. Specifically, whena touch input is applied to the translation screen 703 in FIG. 7D, apre-translation extraction screen 702 and the translation screen 703 maybe saved at the same time.

FIGS. 8A through 8D show examples of providing a result of applying acontact save function to text information extracted from a previewimage.

For example, as illustrated in FIG. 8A, a business card including acontact name and a name may be displayed as a preview image 801.Subsequently, when a text search option 233 is selected, and then atouch input is applied to an area having a business card as shown inFIG. 8B, text information included in the business card is extracted andrecognized. At this time, even when the business card is not exactlyaligned with an outline of the preview image, text information may beextracted and recognized.

In addition, in one example, while the text information included in thebusiness card is extracted and recognized, the preview image 801 itselfmay be captured and stored in the memory 160. In other words, thescanning of text is carried out simultaneously with capturing.

When all of the text information included in the business card isrecognized, an extraction screen 802 is displayed, in which theextracted and recognized business card information is displayed in apredetermined type of business card image. Then, a plurality of searchoptions are switched to function icons 830 related to business cardinformation. A contact information storage icon 835 related to theattributes of business card information displayed on the extractionscreen 802 is included in the function icons 830.

When a touch input is applied to the contact information storage icon835, a contact add screen 8036 is displayed immediately. At this time,business card information that has been displayed on the extractionscreen 802 is automatically entered into the contact item of the contactadd screen 8036.

In addition, the present disclosure may be applicable not only to thepreview image, but also to an image stored in the gallery application,and thus the user may only capture a business card or document and thenapply a text search option at his or her desired time.

Meanwhile, FIGS. 9A through 9D show examples of other functions relatedto a text search option.

FIG. 9A provides an icon for storage at the bottom of the translationscreen 902 for the extracted text information, for example, a clipboardtype of storage icon 910. Here, when a touch input is applied to thestorage icon 910, only the content of the translation screen 902 isstored in a clipboard separately from the extracted text information.Through this, it may be possible to process the extraction screen 901and the translation screen 902 separately.

FIG. 9B shows an example of selecting some text information from theextraction screen 803 including business card information to immediatelyapply it to the execution of its related application.

Specifically, when a touch input is selectively applied only to “name”and “phone number” in the business card information of the extractionscreen 803 illustrated in FIG. 9B, application icons 920 such ashyperlinked phone application, contacts application, share applicationfor exporting contacts may be provided at the bottom of the displaymodule 151 a to immediately execute it with only the selectedinformation.

For an example of another function, as shown in FIG. 9C, when sometranslation languages are selected through a touch input on atranslation screen 904 for text information included in the extractionscreen 901, sub-function icons 940 related to the processing of theselected some translation languages may be additionally provided.

Furthermore, for another example, while a first group of function iconsrelated to the processing of text information included in the extractionscreen are displayed, as illustrated in FIG. 9D, the first group offunction icons may be changed to a second group of function iconsrelated to the attribute of some selected text in response to theselection of some text (e.g., “heart” (text1), “eye” (text2) in theextraction screen 901.

Specifically, in FIG. 9D, it may be seen that a web search function icon933 related to text1 and text2 is provided as a user interface 930(a).When a touch input is applied to the web search function icon 933, theweb search results of text1 and text2 are displayed in an applicationscreen 960, and image information 961, 962 for text1 and text2 are alsodisplayed in the application screen 960(b).

On the other hand, in a text search option, it may be possible todesignate a specific text region to correct it in a desired form. FIGS.10A and 10B show an example related to this.

Specifically, as illustrated in FIG. 10A, even after the extractionscreen 1001 including the extracted text information is displayed, and atranslation function icon 1034 is selected in connection to theprocessing of the text information, an execution icon 1010 for adjustingan outline of the extraction screen may be provided at an upper end ofthe display module 151.

When a touch input is applied to the execution icon 1010 for adjustingthe outline of the extraction screen, a preview image screen isdisplayed again as illustrated in FIG. 10B. In addition, a guide objectfor adjusting the outline is displayed in a text region corresponding tothe extraction screen in the preview screen. The user may apply a dragtouch input to the corner points (P1, P2) of the guide object accordingto the guide of the displayed guide information 1020 to adjust theoutline, that is a size of the crop area, of the extraction screen.

When the outline is adjusted in this way, the amount of text informationincluded in the extraction screen is also automatically edited.Furthermore, the saved image may be varied or additionally saved tocorrespond to the changed outline.

Hereinafter, FIGS. 11A, 11B and 11C show an example of an operation ofexecuting a QR scan function on an image of a subject in a mobileterminal associated with the present disclosure.

First, referring to FIG. 11A, an object including a QR code is pointedon a preview image 1101 according to the execution of the camera, and aQR search option 234 selected in the user interface 230 including aplurality of search options is recognized.

Then, as illustrated in FIG. 11B, the preview image 1101 is switched toa screen 1102 on which a guide object for improving the recognition rateof the QR code is displayed. Here, the process of touching an object toselect it may be omitted.

When the QR code included in the guide object is recognized, the screenis switched to a page 1103 corresponding to the recognized QR code asillustrated in FIG. 11C. Here, when a back control command is executed,the page 1103 may be quickly switched back to the screen 1102 of FIG.11B.

As described above, according to a mobile terminal and a control methodthereof according to an embodiment of the present disclosure, a user mayspecify a search target by simply pointing a camera without having todirectly perform information search related to an object in reality, andquickly and easily acquire his or her desired information search result.Furthermore, a desired option may be accurately specified among aplurality of search options prior to image analysis on a subject,thereby eliminating errors due to automatic selection of a search objectand a search option. In addition, a desired search result may beacquired by selecting a desired search option from a preview imageincluding a plurality of objects, and then selectively specifying anyone object from the beginning or including all of the objects. Moreover,it may be possible to quickly change a target object or a search optionitself even after checking the search result.

The foregoing present disclosure may be implemented as codes readable bya computer on a medium written by the program. The computer-readablemedia may include all kinds of recording devices in which data readableby a computer system is stored. Examples of the computer-readable mediamay include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and opticaldata storage device, and the like, and also include a device implementedin the form of a carrier wave (for example, transmission via theInternet). In addition, the computer may include the controller 180 ofthe mobile terminal. Accordingly, the detailed description thereofshould not be construed as restrictive in all aspects but considered asillustrative. The scope of the invention should be determined byreasonable interpretation of the appended claims and all changes thatcome within the equivalent scope of the invention are included in thescope of the invention.

What is claimed is:
 1. A mobile terminal comprising: a camera providedin a main body; a display module configured to display a preview imagefor a shooting of the camera; and a controller configured to detect anentry of a subject into an information search mode while the previewimage is being displayed, and detect a selection of a search option of aplurality of search options for the subject in the information searchmode, wherein the controller is further configured to capture thesubject including an object to which the selected search option is to beapplied, and display an image of the captured subject along with anapplication screen of the selected search option on the display module.2. The mobile terminal of claim 1, wherein: a plurality of itemscorresponding to an execution result of the selected search option forthe object are displayed on the application screen of the selectedsearch option; and in response to a selection of an item of thedisplayed plurality of items, a page screen linked to the selected itemis displayed.
 3. The mobile terminal of claim 1, wherein when a touchinput is applied to a background region of the preview image subsequentto the selection of the search option, the selected search option isapplied to all objects included in the preview image.
 4. The mobileterminal of claim 1, wherein when a back execution command is receivedwhile the application screen of the selected search option is displayed,the application screen is switched to a screen for changing an object towhich the selected search option is to be applied.
 5. The mobileterminal of claim 1, wherein, when, subsequent to the selection of thesearch option, a touch input is applied to any one object of a pluralityof objects included in the preview image, the controller is furtherconfigured to limit application of the selected search option toremaining objects of the plurality of objects.
 6. The mobile terminal ofclaim 5, wherein while the image of the captured subject is displayedalong with the application screen of the selected search option, a guideimage indicating the object to which the selected search option is to beapplied is displayed on the image of the captured subject.
 7. A mobileterminal comprising: a camera provided in a main body; a display moduleconfigured to display a preview image for a shooting of the camera; anda controller configured to detect an entry of a subject into aninformation search mode while the preview image is being displayed, anddetect a selection of a text search option among a plurality of searchoptions for the subject in the information search mode, wherein thecontroller is further configured to extract text information to whichthe text search option is to be applied from the preview image based ona touch input applied to the preview image, switch the preview image toan extraction screen displayed with the extracted text information, andswitch display of the plurality of search options to display of afunction icon related to processing of the text information displayed onthe extraction screen.
 8. The mobile terminal of claim 7, wherein thefunction icon related to the processing of the text information isdetermined differently according to an attribute of the text informationdisplayed on the extraction screen.
 9. The mobile terminal of claim 8,wherein, in response to a selection of a function icon of a plurality offunction icons related to the processing of the text information, anexecution screen of a function corresponding to the selected functionicon for the text information is displayed on the display module. 10.The mobile terminal of claim 9, wherein when a touch input is applied tothe execution screen, information displayed on the execution screen issaved along with the extraction screen.
 11. The mobile terminal of claim7, wherein in response to a selection of a portion of the textinformation in the extraction screen while a first group of functionicons are being displayed, the first group of function icons are changedto a second group of function icons related to an attribute of theselected portion of the text information.
 12. The mobile terminal ofclaim 7, wherein a capture image of the preview image is stored in amemory while the text information is extracted from the preview imageaccording to the touch input applied to the preview image.